Feature: Integrity check for maven, npm and pypi #261
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
VithikaS
reviewed
Aug 11, 2023
src/main/java/org/dependencytrack/model/IntegrityAnalysisComponent.java
Outdated
Show resolved
Hide resolved
VithikaS
reviewed
Aug 11, 2023
VithikaS
reviewed
Aug 11, 2023
src/test/java/org/dependencytrack/event/kafka/KafkaStreamsTopologyTest.java
Outdated
Show resolved
Hide resolved
VithikaS
reviewed
Aug 11, 2023
src/test/java/org/dependencytrack/event/kafka/processor/IntegrityAnalyzerProcessorTest.java
Outdated
Show resolved
Hide resolved
VithikaS
reviewed
Aug 11, 2023
src/main/java/org/dependencytrack/event/kafka/KafkaEventDispatcher.java
Outdated
Show resolved
Hide resolved
VithikaS
reviewed
Aug 11, 2023
src/main/java/org/dependencytrack/event/kafka/processor/IntegrityAnalysiResultProcessor.java
Outdated
Show resolved
Hide resolved
VithikaS
reviewed
Aug 11, 2023
src/main/java/org/dependencytrack/event/kafka/processor/IntegrityAnalysiResultProcessor.java
Outdated
Show resolved
Hide resolved
VithikaS
reviewed
Aug 11, 2023
src/main/java/org/dependencytrack/persistence/ComponentIntegrityQueryManager.java
Outdated
Show resolved
Hide resolved
VithikaS
reviewed
Aug 11, 2023
VithikaS
reviewed
Aug 11, 2023
nscuro
reviewed
Aug 14, 2023
src/main/java/org/dependencytrack/event/kafka/processor/IntegrityAnalysisResultProcessor.java
Outdated
Show resolved
Hide resolved
src/main/java/org/dependencytrack/event/kafka/processor/IntegrityAnalysisResultProcessor.java
Show resolved
Hide resolved
src/main/java/org/dependencytrack/event/kafka/processor/IntegrityAnalysisResultProcessor.java
Outdated
Show resolved
Hide resolved
src/main/java/org/dependencytrack/persistence/ComponentIntegrityQueryManager.java
Outdated
Show resolved
Hide resolved
sahibamittal
changed the title
* Add bloated BOM for ingestion performance testing Signed-off-by: nscuro <[email protected]> * Prevent query compilation cache being bypassed for `matchSingleIdentity` queries See DependencyTrack/dependency-track#2540 This also cleans the query from containing weird statements like `(cpe != null && cpe == null)` in case a component does not have a CPE. Signed-off-by: nscuro <[email protected]> * WIP: Improve BOM processing performance Signed-off-by: nscuro <[email protected]> * Handle dependency graph Signed-off-by: nscuro <[email protected]> * Improve dependency graph assembly Instead of using individual bulk UPDATE queries, use setters on persistent components instead. This way we can again make use of batched flushing. Signed-off-by: nscuro <[email protected]> * Completely replace old processing logic Also decompose large processing method into multiple smaller ones, and re-implement notifications. Signed-off-by: nscuro <[email protected]> * Fix not all BOM refs being updated with new component identities Signed-off-by: nscuro <[email protected]> * Be smarter about indexing component identities and BOM refs Also add more documentation Signed-off-by: nscuro <[email protected]> * Reduce logging noise Signed-off-by: nscuro <[email protected]> * Mark new components as such ... via new transient field. Required for compatibility with #217 Signed-off-by: nscuro <[email protected]> * Compatibility with #217 Signed-off-by: nscuro <[email protected]> * Cleanup tests Signed-off-by: nscuro <[email protected]> * Reduce code duplication Signed-off-by: nscuro <[email protected]> * Cleanup; Process services Signed-off-by: nscuro <[email protected]> * Finishing touches 🪄 Signed-off-by: nscuro <[email protected]> * Make flush threshold configurable The optimal value could depend on how beefy the database server is, and how much memory is available to the API server. Signed-off-by: nscuro <[email protected]> * Clarify `warn` log when rolling back active transactions Signed-off-by: nscuro <[email protected]> * Log number of consumed components and services before and after de-dupe Signed-off-by: nscuro <[email protected]> * Extend BOM processing test with bloated BOM Signed-off-by: nscuro <[email protected]> * Make component identity matching strict To address DependencyTrack/dependency-track#2519 (comment). Also add regression test for this specific issue. Signed-off-by: nscuro <[email protected]> * Add regression test for DependencyTrack/dependency-track#1905 Signed-off-by: nscuro <[email protected]> * Clarify why "reachability on commit" is disabled; Add assertion for persistent object state Signed-off-by: nscuro <[email protected]> * Add tests for `equals` and `hashCode` of `ComponentIdentity` Signed-off-by: nscuro <[email protected]> * Address review comments Signed-off-by: nscuro <[email protected]> --------- Signed-off-by: nscuro <[email protected]> Signed-off-by: mehab <[email protected]>
* added stack trace Signed-off-by: mehab <[email protected]> --------- Signed-off-by: mehab <[email protected]>
Bumps debian from bullseye-20230612-slim to bullseye-20230703-slim. --- updated-dependencies: - dependency-name: debian dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: mehab <[email protected]>
Bumps [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action) from 2.8.0 to 2.9.0. - [Release notes](https://github.com/docker/setup-buildx-action/releases) - [Commits](docker/setup-buildx-action@v2.8.0...v2.9.0) --- updated-dependencies: - dependency-name: docker/setup-buildx-action dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: mehab <[email protected]>
Signed-off-by: mehab <[email protected]>
Method signatures accepting a `boolean commitIndex` flag have not been modified to avoid a larger refactoring. Invocations of index updates via `Event.dispatch(new IndexEvent(...))` are just commented out so that the information of *when* those updates are supposed to happen is not lost when we re-implement indexing again. Closes DependencyTrack/hyades#661 Signed-off-by: nscuro <[email protected]> Signed-off-by: mehab <[email protected]>
Ported from DependencyTrack/dependency-track#2873 Co-authored-by: shawyeok <[email protected]> Signed-off-by: nscuro <[email protected]> Signed-off-by: mehab <[email protected]>
Ported from DependencyTrack/dependency-track#2872 Co-authored-by: shawyeok <[email protected]> Signed-off-by: nscuro <[email protected]> Signed-off-by: mehab <[email protected]>
Signed-off-by: mehab <[email protected]>
Signed-off-by: mehab <[email protected]>
Signed-off-by: mehab <[email protected]>
Signed-off-by: nscuro <[email protected]> Signed-off-by: mehab <[email protected]>
Supersedes #287 Signed-off-by: nscuro <[email protected]> Signed-off-by: mehab <[email protected]>
Signed-off-by: mehab <[email protected]>
Bumps [actions/checkout](https://github.com/actions/checkout) from 3.6.0 to 4.0.0. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v3.6.0...v4.0.0) --- updated-dependencies: - dependency-name: actions/checkout dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: mehab <[email protected]>
Bumps [aquasecurity/trivy-action](https://github.com/aquasecurity/trivy-action) from 0.11.2 to 0.12.0. - [Release notes](https://github.com/aquasecurity/trivy-action/releases) - [Commits](aquasecurity/trivy-action@0.11.2...0.12.0) --- updated-dependencies: - dependency-name: aquasecurity/trivy-action dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: mehab <[email protected]>
…nerability analysis completes Signed-off-by: nscuro <[email protected]>
Bumps [com.github.tomakehurst:wiremock-jre8](https://github.com/wiremock/wiremock) from 2.35.0 to 2.35.1. - [Release notes](https://github.com/wiremock/wiremock/releases) - [Commits](wiremock/wiremock@2.35.0...2.35.1) --- updated-dependencies: - dependency-name: com.github.tomakehurst:wiremock-jre8 dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: mehab <[email protected]>
This field is displayed in the UI. Despite the variable name only mentioning "format", it previously also included the spec version (https://github.com/DependencyTrack/dependency-track/blob/939f3e83875b4a6a8ab73e82fbf973f00ea993d3/src/main/java/org/dependencytrack/tasks/BomUploadProcessingTask.java#L157). This was missed when refactoring the BOM ingestion. Signed-off-by: nscuro <[email protected]> Signed-off-by: mehab <[email protected]>
Bumps `lib.protobuf-java.version` from 3.24.2 to 3.24.3. Updates `com.google.protobuf:protobuf-java` from 3.24.2 to 3.24.3 - [Release notes](https://github.com/protocolbuffers/protobuf/releases) - [Changelog](https://github.com/protocolbuffers/protobuf/blob/main/protobuf_release.bzl) - [Commits](protocolbuffers/protobuf@v3.24.2...v3.24.3) Updates `com.google.protobuf:protobuf-java-util` from 3.24.2 to 3.24.3 --- updated-dependencies: - dependency-name: com.google.protobuf:protobuf-java dependency-type: direct:production update-type: version-update:semver-patch - dependency-name: com.google.protobuf:protobuf-java-util dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: mehab <[email protected]>
Bumps org.slf4j:log4j-over-slf4j from 2.0.7 to 2.0.9. --- updated-dependencies: - dependency-name: org.slf4j:log4j-over-slf4j dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: mehab <[email protected]>
Signed-off-by: vithikashukla <[email protected]> Signed-off-by: mehab <[email protected]>
Co-authored-by: Niklas <[email protected]> Signed-off-by: VithikaS <[email protected]> Signed-off-by: mehab <[email protected]>
Signed-off-by: mehab <[email protected]>
Bumps debian from bullseye-20230814-slim to bullseye-20230904-slim. --- updated-dependencies: - dependency-name: debian dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: mehab <[email protected]>
Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 3.1.2 to 3.1.3. - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](actions/upload-artifact@v3.1.2...v3.1.3) --- updated-dependencies: - dependency-name: actions/upload-artifact dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: mehab <[email protected]>
Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 4.1.1 to 4.2.1. - [Release notes](https://github.com/docker/build-push-action/releases) - [Commits](docker/build-push-action@v4.1.1...v4.2.1) --- updated-dependencies: - dependency-name: docker/build-push-action dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: mehab <[email protected]>
Bumps org.apache.commons:commons-compress from 1.23.0 to 1.24.0. --- updated-dependencies: - dependency-name: org.apache.commons:commons-compress dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: mehab <[email protected]>
Signed-off-by: mehab <[email protected]>
Signed-off-by: mehab <[email protected]>
Signed-off-by: mehab <[email protected]>
Signed-off-by: mehab <[email protected]>
Signed-off-by: mehab <[email protected]>
Signed-off-by: mehab <[email protected]>
Signed-off-by: mehab <[email protected]>
mehab
force-pushed
the
feature/integritycheckmaven
branch
from
d1b8f66 to
7947e6e
Compare
Signed-off-by: mehab <[email protected]>
|
Closing this pull request after reconsidering the design as per requirement of having published date for newly fetched components all the time. Adding the detailed meeting notes on issue DependencyTrack/hyades#699 |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
The integrity check functionality (maven, npm and pypi).
Addressed Issue
DependencyTrack/hyades#699
Checklist